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Background: Responding to mandates from the Accreditation Council for Graduate Medical Education 
(ACGME) and American Osteopathic Association (AOA), residency programs have developed competency- 
based assessment tools. One such tool is the American College of Osteopathic Pediatricians (ACOP) program 
directors' annual report. High-stakes clinical skills licensing examinations, such as the Comprehensive 
Osteopathic Medical Licensing Examination Level 2-Performance Evaluation (COMLEX-USA Level 2-PE), 
also assess competency in several clinical domains. 

Objective: The purpose of this study is to investigate the relationships between program director competency 
ratings of first-year osteopathic residents in pediatrics and COMLEX-USA Level 2-PE scores from 2005 to 2009. 
Methods: The sample included all 94 pediatric first-year residents who took COMLEX-USA Level 2-PE and 
whose training was reviewed by the ACOP for approval of training between 2005 and 2009. Program director 
competency ratings and COMLEX-USA Level 2-PE scores (domain and component) were merged and analyzed 
for relationships. 

Results: Biomedical/biomechanical domain scores were positively correlated with overall program director 
competency ratings. Humanistic domain scores were not significantly correlated with overall program director 
competency ratings, but did show moderate correlation with ratings for interpersonal and communication skills. 
The six ACGME or seven AOA competencies assessed empirically by the ACOP program directors' annual 
report could not be recovered by principal component analysis; instead, three factors were identified, accounting 
for 86% of the variance between competency ratings. 

Discussion: A few significant correlations were noted between COMLEX-USA Level 2-PE scores and program 
director competency ratings. Exploring relationships between different clinical skills assessments is inherently 
difficult because of the heterogeneity of tools used and overlap of constructs within the AOA and ACGME core 
competencies. 
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Like residency training programs accredited by the 
Accreditation Council for Graduate Medical 
Education (ACGME), American Osteopathic As- 
sociation (AOA) accredited residency programs are 
required to assess each of the seven core osteopathic 
competencies (osteopathic philosophy and osteopathic 
manipulative medicine, medical knowledge, patient care, 



interpersonal and communication skills, professionalism, 
practice-based learning and improvement, and systems- 
based practice) for all residents in each year of training 
(1, 2). These competencies are comparable to the six core 
competencies of the ACGME (2). 

Program director ratings and annual evaluations have 
been used to evaluate resident performance (3-8). Since 
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2005 osteopathic pediatric program directors have been 
rating resident performance using the American College 
of Osteopathic Pediatricians (ACOP) program directors' 
annual report. This competency-based global assessment 
tool is completed annually and used to evaluate resident 
progress throughout training. Independent competency 
ratings are strongly related to each other (9). 

The Comprehensive Osteopathic Medical Licen- 
sing Examination Level 2-Performance Evaluation 
(COMLEX-USA Level 2-PE) is a high-stakes licensure 
exam based on standardized patients (SPs) and used to 
assess clinical skills competency of osteopathic medical 
students entering graduate medical education ( 1 0) . For this 
exam, examinees rotate through 12 stations in which they 
evaluate SPs who have been trained to simulate a variety of 
typical clinical presentations. The exam specifically mea- 
sures a variety of clinical skills, including doctor-patient 
communication, interpersonal skills and professionalism; 
medical history taking and physical examination skills; 
osteopathic principles and manipulative treatment; and 
written communication skills. COMLEX-USA Level 2-PE 
clinical skill component scores have good internal consis- 
tency (10-12). Furthermore, previous study of first-year 
osteopathic pediatric residents showed overall program 
director competency ratings were positively correlated with 
COMLEX-USA cognitive exam scores (Levels 1-3) as well 
as biomedical/biomechanical domain scores of the clinical 
skills exam, COMLEX-USA Level 2-PE (9). 

COMLEX-USA Level 2-PE measures fundamental 
clinical skill competence of medical students in the 
context of SP encounters. The ACOP program directors' 
annual report measures each of the seven competencies of 
pediatric residents in the context of overall performance 
throughout each year of residency training. Both assess- 
ment tools seem to measure common clinical skills, but in 
different settings, formats and times during medical 
training. Do they truly measure similar skills? For 
instance, both tools measure an examinee's ability to 
communicate with patients; therefore, one would expect 
communication skills ratings of the ACOP program 
directors' annual report and COMLEX-USA Level 2- 
PE to be highly correlated. The purpose of this study is to 
investigate relationships between COMLEX-USA Level 
2-PE clinical skill scores and competency-based global 
assessments of pediatric residents as reported by osteo- 
pathic residency program directors. In particular, we 
investigate the extent to which domain and individual 
component scores of COMLEX-USA Level 2-PE are 
related to the individual program director competency 
ratings of medical knowledge, patient care, interpersonal 
and communication skills, professionalism, practice- 
based learning and improvement, and systems-based 
practice of first-year pediatric residents. 



Methods 

Measures 

COMLEX-USA Level 2-PE clinical skill scores 
COMLEX-USA Level 2-PE is a 12-station SP-based 
high-stakes clinical skills examination used for licensing 
osteopathic physicians, where examinees interview and 
examine SPs who are trained to simulate different 
medical complaints (10). Each station provides 14 
minutes for an examinee to interview the SP, followed 
by a nine-minute post-encounter period of time to 
complete the written patient note (subjective/objective/ 
assessment/plan - SOAP note) summarizing the encoun- 
ter. Cases vary according to presenting symptoms 
(respiratory, cardiovascular, neuromusculoskeletal, gas- 
trointestinal and other symptoms), age, gender and race/ 
ethnicity. Cases also vary in clinical complaints that could 
be acute, chronic or provide opportunities for health 
promotion or disease prevention. 

Examinees, usually medical students toward the end of 
their third year or beginning of their fourth year of 
training, must pass two domains (humanistic and bio- 
medical/biomechanical) to pass the examination. Pass/fail 
decisions based on the standard of minimal competence 
for entry into graduate medical education (10) are 
operationalized by each domain's cut score. Examinee 
performance is scored using several clinical skill measures 
within each domain (Table 1). 

The humanistic domain comprises scores to assess the 
examinee's listening skills, respectfulness, empathy, pro- 
fessionalism, ability to elicit information and ability to 
provide information. Each of these components is scored 
by the SP using the global patient assessment tool, a 
holistic Likert-type scoring instrument (10). As a measure 
of reliability, the generalizability coefficients for huma- 
nistic domain scores are comparable to those of other 
high-stakes clinical skills examinations used for licensure 
(10, 11). 

The biomedical/biomechanical domain comprises three 
weighted component scores: data gathering (ability to 
obtain a medical history and perform a physical exam); 
written patient note (ability to complete a written commu- 
nication task and synthesize information, develop a 
differential diagnosis and formulate a diagnostic and 
treatment plan); and osteopathic manipulative treatment 
(OMT: ability to integrate osteopathic principles and use 
OMT) (10). Data gathering scores are measured on a 
percentage metric based on the number of history 
questions and physical examination maneuvers correctly 
performed during the clinical encounter. Written patient 
notes are scored by trained physician examiners using a 
Likert-type holistic instrument. OMT skills, specifically 
scored in 25-40% of encounters, are scored by trained 
physician examiners using a different Likert-type holistic 
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Table 1. COMLEX-USA Level 2-PE domains and components 



Domain Component 



Humanistic Global patient assessment tool (including assessment of professionalism, respect, eliciting 

information, giving information, empathy and listening) 
Biomedical/biomechanical Data gathering (history and physical) Written patient note (SOAP note) OMT 9 

a Although osteopathic principles are assessed throughout the examination, OMT is specifically scored in 25^1-0% of all encounters. 



tool. Generalizability coefficients for biomedical/ 
biomechanical domain scores are comparable to those of 
other high-stakes clinical skills examinations (10). 

Component scores for patient encounters are averaged 
across all scored stations to compute average perfor- 
mance during the testing day. After scores are equated for 
rater leniency or stringency, a pass/fail decision is 
rendered for each candidate. Pass/fail determinations 
are based on standards applied to the cut scores of 
each domain using an examinee-centered standard setting 
method (10). Only pass/fail scores, not component scores, 
are reported to examinees, school deans and residency 
program directors. 

ACOP program directors' annual report: competency 
ratings of residents 

Osteopathic pediatric program directors complete 
the ACOP program directors' annual report for each 
resident at the end of each year of training. This global 
assessment is used to address performance related to 
resident scholarly and research activity, continuity patient 
panel size, overall progress and skill assessment (3). For 
each resident, the program director assigns a rating to 
each skill, classified by competency, using a nine-point 
rating scale ranging from 1 (lowest) to 9 (highest). 
Program director ratings are recorded by one director 
from each residency program, thus inter-rater reliability 
measures are not available. The specific skill assessments 
for each of the seven osteopathic medical competencies 
are presented in Table 2. The reports are then forwarded 
to the ACOP Graduate Medical Education Committee 
for review and certification of training. ACOP program 
directors' annual report ratings have been shown to be 
reliable, and independent competency ratings are strongly 
related to each other (9). 

Design 

Data collected from all pediatric osteopathic residents 
reviewed by the ACOP Graduate Medical Education 
Committee from 2005 through 2009 were compiled from 
the ACOP program directors' annual reports. Data 
include competency-based skill assessment ratings for 
each resident. Program director ratings are reviewed as a 



global measure of resident performance by the ACOP 
Graduate Medical Education Committee. Therefore, skill 
assessment ratings within each competency were summed 
together into a total competency score for each of the 
seven core competencies, labeled as program director 
competency ratings for this study. 

A linking variable unknown to the researchers was 
created to match to the National Board of Osteopathic 
Medical Examiners (NBOME) database to extract all 
COMLEX-USA domain and component scores. Results 
of this study are reported in aggregate form to maintain 
confidentiality. Institutional Review Board approval was 
received to analyze these data for this study. 

Sample 

The initial sample consists of 134 residents, representing 
all first-year pediatric residents who were reviewed 
between 2005 and 2009. The sample includes residents 
from 19 pediatric training institutions across the United 
States. 

Of the total sample, 94 first-year residents (69%) took 
the COMLEX-USA Level 2-PE examination, as a 
minority of residents entered a training program prior 
to taking this exam. Most of the 94 first-year residents 
completed COMLEX-USA Level 2-PE toward the be- 
ginning of their fourth year in medical school. 

Analysis 

For each subject (N=94), correlations between 
COMLEX-USA Level 2-PE domain and component 
scores and program director scores for each competency 
were analyzed for statistical comparison using SAS 
version 9.1. The extent to which scores on COMLEX- 
USA Level 2-PE were related to the program director 
competency ratings of medical knowledge, patient care, 
interpersonal and communication skills, professionalism, 
practice-based learning and improvement, and systems- 
based practice were measured in terms of linear depen- 
dency or Pearson correlations (r). Overall domain scores 
(biomedical/biomechanical and humanistic) and indivi- 
dual component scores (OMT, SOAP note, data gather- 
ing and the six skills assessed within the humanistic 
domain) were analyzed for relatedness in conjunction 
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Table 2. Competency-based skill assessment of residents as 
reported using ACOP program directors' annual report 



Competency 



Skill assessment 



Osteopathic philosophy 
and OMT 



Medical knowledge 



Patient care 



Interpersonal and com- 
munication skills 



Professionalism 



Practice-based learning 
and improvement 



Systems-based practice 



1 . Demonstrates competency in 
the understanding and application of 
OMT where appropriate. 

2. Integrates osteopathic concepts 
into the medical care provided to 
patients as appropriate. 

1 . Demonstrates competency in 
the application of clinical medicine to 
patient care. 

2. Knows and applies the founda- 
tions of clinical and behavioral medicine. 

3. Demonstrates knowledge of 
accepted standards of care. 

4. Remains current with new 
developments in medicine. 

5. Participates in lifelong learning 
activities. 

1 . Gathers accurate, essential 
information from all sources. 

2. Demonstrates competency in 
the performance of diagnostic and 
treatment procedures. 

3. Provides healthcare services 
that include preventive medicine and 
health promotion. 

1. Demonstrates effectiveness in 
developing appropriate doctor- 
patient relationships. 

2. Exhibits effective listening, writ- 
ten and oral communication skills. 

1 . Demonstrates respect for 
patients/families and acts as their 
advocate. 

2. Adheres to ethical principles in 
the practice of medicine. 

3. Is sensitive to cultural diversity, 
i.e. religion, age, gender, sexual 
orientation and disabilities. 

4. Is cognizant of his/her own 
physical and mental health in order 
to care effectively for patients. 

1 . Treats patients in a manner 
consistent with current and evi- 
dence-based information. 

2. Performs self-evaluations of 
clinical practice. 

3. Understands research methods, 
medical informatics and the 
application of technology. 

1 . Understands national and local 
health care delivery systems. 

2. Advocates for quality health care 
on behalf of patients. 

3. Practices cost effective medicine. 



with the program director competency ratings. In addi- 
tion, because the program directors' global assessment 
tool has not been evaluated for internal consistency, a 
principal component analysis was performed. This ana- 
lysis was used to account for the variability of the 
individual competency rating scores, and then used as a 
predictor or criterion variable for the subsequent analysis. 

Results 

COMLEX-USA Level 2-PE biomedical/biomechanical 
and humanistic domain clinical skill component scores 
were correlated with program director competency ratings 
to investigate their relationships (Tables 3 and 4). A 
significant correlation was shown between overall biome- 
dical/biomechanical scores and program director compe- 
tency ratings (r=0.29, < .01). In addition, within the 
biomedical/biomechanical domain, SOAP note scores 
showed significant correlations with all competency rat- 
ings (r =0.22-0.29, /><.05), and data gathering scores 
showed significant correlations with program director 
competency ratings for patient care (r=0.23, /><.05) 
and interpersonal and communication skills (r=0.23, 
p<.05). 

COMLEX-USA Level 2-PE humanistic domain scores 
significantly correlated with program director competency 
ratings of interpersonal and communication skills 
(r =0.24, p <.05). Within the humanistic domain, profes- 
sionalism showed significant correlations with compe- 
tency ratings for patient care, interpersonal and 
communication skills, professionalism, practice-based 
learning and improvement, and overall competency 
(r =0.21-0.26, p <.01). Empathy also showed significant 
correlations with program director competency ratings of 
interpersonal and communication skills (r =0.27, p <.01). 

Due to the few significant correlations (between 
individual component scores within the humanistic 
domain and program director competency ratings), a 
principal component analysis was performed on the 
competency rating data. Principal component analysis 
assumptions were tested to determine if the 22 items on 
the ACOP program directors' annual report could be 
optimally combined into one or more smaller composites. 
The assumption that competency items are uncorre- 
cted was rejected by Bartlett's test of sphericity, 
X 2 (231) =4729.83, p <.001. A Kaiser-Meyer-Olkin mea- 
sure of sampling adequacy, 0.95, was sufficiently large to 
proceed with the analysis. Cronbach's coefficient alpha 
showed sufficient internal consistency among compe- 
tency items (0.982). Three principal components 
explained 85.9% of the variance between the 22 compe- 
tency items, with explained variance ranging from 4.7% 
to 73.6%. The principal component analysis was followed 
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Table 3. Pearson correlation (r) matrix of COMLEX-USA Level 2-PE biomedical/biomechanical domain scores and program 
director competency ratings of first-year pediatric residents (N =94) 



Program director competency ratings 





Overall 


OMT 


MK 


PC 


IC 


P 


PBL 


SBP 


Biomedical/biomechanical domain 


















Overall 


.29 b 


.22 a 


.25 a 


.28 b 


.34 b 


.30 b 


.22 a 


.22 a 


OMT 


.14 


.08 


.13 


.11 


.19 


.15 


.10 


.12 


SOAP note 


.29 b 


.28 b 


.23 a 


.29 b 


.27 b 


.27 b 


.28 b 


.22 a 


Data gathering 


.18 


.14 


.14 


.23 a 


.23 a 


.19 


.12 


.12 



Note: MK - medical knowledge; PC - patient care; IC - interpersonal and communication skills; P 
based learning and improvement; SBP - systems-based practice. 
Significant at p <0.05. 
Significant at p <0.01. 



professionalism; PBLI - practice- 



by varimax rotation of the factor matrix. The pattern of 
item regression on factors lacked simplicity for several 
items, indicating the three factors underlying competency 
items are correlated. 

Principal component analysis was also completed for 
each of the competency categories individually (medical 
knowledge, OMT, patient care, interpersonal/communica- 
tion, professionalism, practice-based learning and sys- 
tems-based practice), which yielded a one-component 
solution for each competency, accounting for 90-99% of 
total variance explained. Of the 22 skill assessments of 
the ACOP program directors' annual report, analysis 
identified three underlying factors, which we named as 
existing competency classifications: osteopathic principles 



and treatment and medical knowledge; systems-based 
practice and practice-based learning and improvement; 
and patient care, interpersonal and communication skills, 
and professionalism. The six ACGME or seven 
AOA general competencies assessed empirically by pro- 
gram director ratings could not be recovered with a 
principal component analysis. Instead, the three factors 
were identified to explain 86% of the variance between 
competency ratings. 

Discussion 

Residency program directors routinely evaluate residents 
of their training programs. Investigating how these 
ratings correspond to other measures of clinical compe- 



Table 4. Pearson correlation (r) matrix of COMLEX-USA Level 2-PE humanistic domain scores and program director 
competency ratings of first-year pediatric residents (N = 94) 



Program director competency ratings 





Overall 


OMT 


MK 


PC 


IC 


P 


PBLI 


SBP 


Humanistic domain 


















Overall 


.13 


.10 


.07 


.16 


.24 a 


.19 


.09 


.05 


El 


.19 


.20 


.19 


.18 


.18 


.16 


.19 


.08 


LI 


.04 


.05 


.01 


.05 


.11 


.06 


.05 


-.01 


Gl 


.12 


.12 


.07 


.11 


.16 


.15 


.14 


.08 


Resp 


.12 


.07 


.11 


.14 


.20 


.15 


.05 


.06 


EM 


.17 


.12 


.08 


.19 


.27 b 


.20 


.16 


.11 


PR 


.23 a 


.18 


.18 


.22 a 


.26 a 


.22 a 


.21 a 


.18 



Note: El - eliciting information; LI - listening; Gl - giving information; Resp, respect; EM - empathy; PR - professionalism; MK - medical 
knowledge; PC - patient care; IC - interpersonal and communication skills; P - professionalism; PBLI - practice-based learning and 
improvement; SBP - systems-based practice. 
Significant at p <0.05. 
Significant at p <0.01. 
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tence helps to justify resident evaluation by the program 
director. Overall program director competency ratings 
have been shown to be positively correlated with the 
biomedical/biomechanical domain scores of the clinical 
skills exam, COMLEX-USA Level 2-PE (9). We demon- 
strate in this study that to some extent domain and 
individual component scores of COMLEX-USA Level 2- 
PE are related to the individual program director 
competency ratings of medical knowledge, patient care, 
interpersonal and communication skills, professionalism, 
practice-based learning and improvement, and systems- 
based practice of first-year pediatric residents. 

Clinical skills examination performance, as measured 
by the biomedical/biomechanical domain scores, corre- 
lated positively with overall program director competency 
ratings, exhibiting a significant amount of convergent 
validity. The constructs measured within this domain of 
the COMLEX-USA Level 2-PE and the ACOP program 
director reports are similar, and appear to be measuring 
similar concepts. When completing global assessment 
ratings, such as the ACOP program director reports, 
assessors likely elicit input from many sources and 
faculty. This information, albeit less reliable than first- 
hand accounts, is likely to focus on residents' cognitive 
abilities and medical knowledge, consistent with the 
clinical skills targeted in the biomedical/biomechanical 
domain of COMLEX-USA Level 2-PE. 

Humanistic domain scores also correlated positively 
with program director ratings for interpersonal and 
communication skills. Program directors are often in a 
unique position of directly observing morning reports and 
resident presentations, whereby medical knowledge and 
communication skills are directly evaluated. Because these 
skills are directly observed by program directors, this may 
account for the positive correlations between their com- 
petency ratings (particularly professionalism and commu- 
nication skills) and COMLEX-USA Level 2-PE scores. 

Although significant correlations were found between 
biomedical/biomechanical domain scores and program 
director competency ratings, few significant correlations 
between component scores within the humanistic domain 
and competency ratings were identified. The constructs 
measured within the humanistic domain may be dissim- 
ilar to those measured within the ACOP program director 
reports, as suggested by the low number of significant 
correlations. Alternatively, perhaps the tools are inher- 
ently different and measuring something unique (diver- 
gent validity). Previous studies recognize the complexity 
of interpreting correlations between medical student and 
resident performance given the heterogeneity of assess- 
ment tools being used at different levels of training (13, 
14), and there needs to be a more consistent approach to 
the use of tools for assessing external validity. Perhaps, as 
another explanation, both tools are global assessments, 
and neither is measuring discrete or unique clinical skills. 



Previous studies demonstrate that current measurement 
tools poorly discriminate competencies from one another, 
and competencies cannot be independently assessed (15). 
In addition, the tool used to measure the humanistic 
domain in COMLEX-USA Level 2-PE is generalizable 
and more consistent with a global assessment of huma- 
nistic clinical skills (1 1). Our data suggest that the ACOP 
program directors' annual reports are measuring overall 
resident competence, and one should use caution when 
interpreting scores of individual skill components. 

In a previous study, factor analysis indicated that 
program directors from a variety of subspecialties assess 
competence in two domains: medical knowledge and 
interpersonal skills (8). Another study identified four 
competency clusters: professionalism, patient care, med- 
ical knowledge and communication (16). Using principal 
component analysis, we identified three factors, rather 
than the six ACGME or seven AOA general competen- 
cies. This may confound our results and help to explain 
why more correlations were not found between program 
director competency ratings and COMLEX-USA Level 
2-PE component scores. 

This study has a few limitations. First, although 
correlations found in the study were statistically signifi- 
cant, many were weak and may lack practical significance. 
Further research, such as addressing construct validity of 
the tools and focus group discussions, may help to 
elucidate the practical implications of the significant 
findings. Second, the sample in this study represents all 
first-year pediatric residents reviewed by the ACOP 
Graduate Medical Education Committee between 2005 
and 2009 who completed the COMLEX-USA Level 2-PE; 
therefore, results are limited to this sample and may not be 
applicable to residents of other specialties. The study 
excluded many osteopathic interns who train in tradition- 
ally allopathic programs, where program directors for this 
group of residents did not complete the ACOP program 
directors' annual report. A total of 691 osteopathic 
physicians were enrolled in ACGME-approved pediatric 
residency programs in 2009-2010, representing 8.5% of all 
pediatric residents in allopathic training programs (17), 
and 190 allopathic pediatric residency positions were filled 
through the National Resident Matching Program in 2009 
(18). This suggests that a large number of osteopathic 
pediatric interns, whose training is certified by the 
ACGME and not by the AOA or ACOP, were not 
available for this study. Future study could involve 
assessing all osteopathic pediatric residents by creating a 
common assessment tool used by both AOA- and 
ACGME-accredited residency program directors. 

All residency program directors likely utilize some form 
of global resident assessment, but these vary from institu- 
tion to institution. The ACOP program directors' annual 
report is a common global assessment tool used by 
all osteopathic pediatric program directors across all 
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AOA-accredited residency programs. Although findings of 
this study apply to a limited sample of residents, results 
suggest that global competency-based resident assess- 
ments do correlate with high-stakes SP-based clinical skills 
assessments to some extent. This information may be 
helpful for developing future research protocols for asses- 
sing clinical skills of residents of ACGME-accredited 
programs and residents of disciplines other than pediatrics. 

Conclusions 

Results of this study support the view that program 
director global assessments, like those used by most AOA 
and ACGME-accredited residency training programs, 
correlate with high-stakes clinical skills examination 
performance. Results specifically supported three major 
outcomes. First, biomedical/biomechanical domain and 
SOAP note component scores correlated positively with 
overall program director competency ratings. Second, 
humanistic domain scores correlated with program direc- 
tor competency ratings for interpersonal and communica- 
tion skills. Third, three competency assessment factors 
were identified using principal component analysis, rather 
than the six ACGME or seven AOA general competencies. 
Exploring relationships between different clinical skills 
assessments, such as COMLEX-USA Level 2-PE and 
program director competency ratings, is inherently difficult 
because of the heterogeneity of tools used. Future work 
in utilizing comparable assessment tools would help 
to elucidate some of the findings of this study. 
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